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Abstract 

Background: Currently electronic medical records (EMRs) are implemented in hospitals, because of expected 
benefits for quality and safety of care. However the implementation processes are not unproblematic and are 
slower than needed. Many of the barriers and facilitators of the adoption of EMRs are identified, but the relative 
importance of these factors is still undetermined. This paper quantifies the relative importance of known barriers 
and facilitators of EMR, experienced by the users {i.e., nurses and physicians in hospitals). 

Methods: A discrete choice experiment (DCE) was conducted among physicians and nurses. Participants answered 
ten choice sets containing two scenarios. Each scenario included attributes that were based on previously identified 
barriers in the literature: data entry hardware, technical support, attitude head of department, performance 
feedback, flexibility of interface, and decision support. Mixed Multinomial Logit analysis was used to determine the 
relative importance of the attributes. 

Results: Data on 148 nurses and 150 physicians showed that high flexibility of the interface was the factor with 
highest relative importance in their preference to use an EMR. For nurses this attribute was followed by support 
from the head of department, presence of performance feedback from the EMR and presence of decisions support. 
While for physicians this ordering was different: presence of decision support was relatively more important than 
performance feedback and support from the head of department. 

Conclusion: Considering the prominent wish of all the intended users for a flexible interface, currently used EMRs 
only partially comply with the needs of the users, indicating the need for closer incorporation of user needs during 
development stages of EMRs. The differences in priorities amongst nurses and physicians show that different users 
have different needs during the implementation of innovations. Hospital management may use this information to 
design implementation trajectories to fit the needs of various user groups. 

Keywords: Electronic Medical Record, Implementation, Discrete choice experiment, Adopter preferences, Barriers 
and facilitators, Clinicians 
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Background 

Diffusion of innovations results from the interaction be- 
tween technology producers, users, and external groups 
and the system adopting an innovation [1,2]. Although 
user acceptance of an innovation varies between individ- 
uals, studies found several barriers and facilitators that 
generally apply to the implementation of innovations 
[2-6]. Knowledge about these barriers and facilitator is 
integrated in several overlapping theoretical frameworks 
[7]. Barriers and facilitators of implementation in organi- 
zations can be found on the user level (e.g., preferences 
and skills), the organizational level (practical support and 
culture), or on the level of the innovation (ease of use, 
added value to users) [2,3,6]. 

Clearly, a well-designed implementation plan uses fa- 
cilitators and overcomes barriers in the different phases 
of the implementation process. The process begins with 
the proposition to initiate change, moves to evaluation 
and re-adjustment of the implementation plan, then needs 
to involve aspects of the innovation/change, of the stake- 
holders and of the (organizational) context of the imple- 
mentation process [7]. 

In this paper we focus on one phase in this implementa- 
tion process of a specific innovation: the user acceptance 
of electronic medical records (EMRs) in Dutch hospitals. 
Since EMRs are currently implemented or improved in a 
majority of Dutch hospitals, they provide a unique oppor- 
tunity to evaluate the preferences of users. We will use 
their responses to investigate and order their preferences. 
Building on knowledge from prior implementation stud- 
ies, we aim to gain insight in the relative importance of 
the added value of the innovation, the ease of use of the 
innovation and the organizational support in the support 
for the innovation by its users [2,6,8]. 

Quantifying the relative importance of those factors, in 
the preference of users, enables identification of the most 
prominent problems that users face in the current stage of 
the implementation process. This knowledge may help 
developers, implementers, and policy makers to design 
and adapt implementation trajectories, giving attention 
to the most relevant issues as perceived by the users. 

We test the relative importance of these factors using 
a discrete choice experiment (DCE). A DCE is a quanti- 
tative method of eliciting preferences concerning a good 
or service, which is an up-coming method in public 
health to determine what factors influence people's will- 
ingness to participate in and pay for medical or preventive 
interventions [9-11]. 

Two articles [12,13] used conjoint or DCE techniques 
to address implementation related issues. The first study 
used conjoint analyses to elicit preferences of stakeholders 
concerning innovations in general. That study concluded 
that a conjoint analysis is useful in studying the prefer- 
ences of health care professionals [12]. The second study 



used a DCE a to prioritize barriers among physicians for 
the use of a breast cancer guideline. That study concluded 
that it was far too time consuming to let physicians 
prioritize all options [13]. Given the contradicting conclu- 
sions of these studies, the current study has a complemen- 
tary aim to explore the use of DCEs in implementation 
studies amongst health care professionals. 

To realize the potential benefits (eg in communication, 
patient management, research and patient safety [14]; 
structuring the health care process, health care costs reduc- 
tion [15], and improvement of quality [16], efficiency and 
safety [17]) clinicians need to incorporate EMR usage in 
their daily practice. The more accurate, concise and timely 
the patient data are entered into the system, the more 
beneficial the system will be for all its users. Since physi- 
cians and nurses showed positive attitudes towards the 
EMR after using it for a while [17], nothing seems to slow 
down the implementation of this innovation. However, des- 
pite that some studies show positive attitudes and expecta- 
tions, the speed of the adoption remains slow [16,18] and 
the main cause for this slow speed is not always clear. 

A systematic ordering of factors investigated in imple- 
mentation studies reveals that most studies investigate fac- 
tors on the user level, the organizational level or on the 
level of the innovation (Figure 1) [3]. The structural level 
and patient-level factors were studied to a lesser extent. 
Instead of taking all these factors into account, the present 
study focuses on the relative importance of two of the 
most studied levels, namely the innovation and the 
organizational. This choice is not intended to downplay 
the importance of other factors, since social, cultural, and 
political factors are expected to play a role in the EMR im- 
plementation process, but here we aim at investigating the 
two core factors in more detail than earlier studies. While 
they have been studies before, their relative importance 
remained unknown. The second reason to focus on these 
aspects is that in the case of the EMR, the social, and 
political factors are to a large extent similar to all Dutch 
employees, making it impossible to include in this study. 
Third, patient level factors are not taken into account 
because we are mainly interested in the preferences of 
the users {i.e., clinicians) who apply an EMR in the care 
for all patients, and patients do not use EMRs. 

The research question is: What is the relative import- 
ance in the preference of users of EMRs in hospitals of 
factors that are related to the innovation itself (perceived 
benefits and ease of use) and organizational context 
(support from other organizational departments) for 
users (physicians and nurses) of EMRs in hospitals? 

Methods 

Discrete choice experiment 

We used a Discrete Choice Experiment (DCE) to quantify 
the relative importance of the selected factors for the 
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Figure 1 A multi-level framework predicting implementation outcomes [3]. 



willingness to use the EMR of two users groups, physi- 
cians and nurses in Dutch hospitals. DCE is a method to 
elicit individuals' preferences concerning a service or good 
[9,10,19,20]. In a DCE respondents are asked to react on a 
presented scenario. In the current study, we selected re- 
spondents who have experience with working with an 
EMR in their current position. Therefore, little introduc- 
tion to the scenario was necessary. The question prior to 
the presentation of the scenarios was: 'Please indicate in 
which of the following situations you prefer to use the 
electronic medical record'. 

Subsequently, a number of choice sets of two scenarios 
are presented to the respondents. The scenarios are de- 
scribed by different characteristics of the service or good 
(i.e., attributes). Every attribute contains several levels, 
representing different values. In this study the attributes 
are the factors that we expect to affect the willingness to 
use the EMR, and the levels are varied. 

Respondents choose the scenarios based on the differ- 
ences in the levels of the attributes. The DCE is based on 
Random Utility Theory, which assumes that respondents 
choose the scenario with the highest personal utility 
[9,20], whereby utility U exists of a predictable component 
V and an unpredictable component e (U = V + e) [21]. The 
answers of the respondents to multiple choice sets enables 
researchers to estimate the relative importance of different 
attributes of the scenarios [10,11,20]. 

DCE design 

NGene 1.1 software (Choice Metrics, 2011) was used to 
construct the choice sets of this DCE. A D-efficient design 
was used for this procedure [22]. This results in a minimal 
number of choice sets that have to be completed by a min- 
imal number or respondents in order to be able to detect 
significant differences. 

In total, 10 unique choice sets were constructed for 
this DCE (see Additional file 1 for a choice set example). 
Each choice set consisted of two scenarios, subsequently 



asking the respondents: 'Please indicate in which situ- 
ation you prefer to use the electronic medical record' 
After choosing one of the scenarios respondents were 
asked to indicate whether they would indeed use this 
type of EMR or would rather not use it (opt-out). We 
chose this method to measure the opt-out for a number 
of reasons. First, the use of an EMR in a hospital is not a 
real choice for the clinicians. So including an opt-out 
would not be congruent with the real life situation. Sec- 
ond, we wanted to know how willing the respondents 
where to use an EMR per se. By adding the additional 
question, we gathered information on the willingness of 
the respondents to use their EMR. Third, this way of 
measuring the opt-out has the advantage that the infor- 
mation of the trade offs are measured (this information 
is lost in a conventional opt-out) and we have information 
whether the respondents would rather opt-out. Before the 
choice sets were presented to the respondents, all attri- 
butes and levels were explained and at any choice set re- 
spondents had the possibility to see the explanations once 
more (Additional file 2). Additionally, it was explained 
how the choice sets should be answered, including an 
example. 

Attributes and levels 

In the design of a DCE, it is crucial to find a balance be- 
tween a realistic scenario that includes the relevant factors 
and the complexity of the design, to avoid cognitive over- 
load of the respondents. We therefore limited the number 
of attributes to six per scenario. Three of the attributes are 
on the level of the innovation and three are on the level of 
the organization. 

The attributes were based on existing literature and pre- 
tested in short interviews with users of EMRs [2,5,6,23]. 
Subsequently the DCE was piloted among fifteen nurses 
and doctors. It was tested whether they understood the 
questions and if they could relate the scenarios to their 
own situation. This did not lead to changes in the design. 
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Attributes on the innovation level 

On the level of the innovation, we included three attri- 
butes, Data entry hardware, flexibility of the interface and 
the presence of decision support. 

The attribute data entry hardware contains two attri- 
bute levels: 

Computer/workstation: You are able to enter and 
request patient data on a number of fixed locations (e.g., 
at outpatient clinic, operating theater and your office). 

Tablet: You are able to walk in the hospital and update 
and request patient data at any desired location. 

This attribute was selected because, ease of use and 
added value to the user are expected to positively affect 
the willingness to use an innovation [8,23-26]. If EMR 
systems were user-friendly, and compatible with the 
working environment, users perceived them as easy to 
use and valuable to facilitate work processes and were 
more willing to implement the system than when the 
system was unclear and less compatible to the work 
environment [4,27]. 

The levels were chosen to vary the ease of use for the 
users. Recent studies have shown positive attitudes of 
physicians towards the use of tablets, because of its sim- 
plicity, similarity to paper and intuitive nature [28,29]. 
Tablets have shown to enhance interactions between 
physicians and patients and improve their workflow be- 
cause information is easier accessed and physicians have 
increased mobility [28]. 

Flexibility of the interface of the EMR is related the 
ease of use of the innovation. We varied the following 
levels: 

Low: The system provides a comprehensive overview 
of all medical information. You have to scroll to see rele- 
vant information. 

High: You can configure what information is displayed 
at first (e.g., patient data, lab, medication, imaging, diag- 
nosis etc). 

This attribute can also be linked to and efficacy (com- 
patibility) of the innovation [2]. A study indicates that 
nurses were frustrated when they did not gain useful 
data from an EMR system, while they knew that EMR 
systems could assist them with patient care by provid- 
ing in rightful information [30]. Systematic reviews 
show evidence for that when the EMR is not adapted to 
needs or abilities of its users, the EMR is perceived as 
difficult to use [27], causing a user to not adopt the 
EMR [18]. 

The third attribute on the level of the innovation is 
the presence of decision support in the EMR. The two 
attribute levels are: 

Absent: the system has no alarm for risk situations. 

Present: with sound and image you will be warned for 
risky situations (e.g., extreme medication dosage or po- 
tential medication interactions). 



Decision support is designed to aid clinicians to avoid 
mistakes and may therefore improve job performance 
[23] and as such have added value in terms of product- 
ivity and efficiency [27] for the users, increasing the 
chance of adoption [4]. The decision support system 
can alert or remind users in case of adverse drug events, 
for example overmedication or antagonizing of different 
drugs. The reminders and alerts can warn a user when 
changes are made in a patient's EMR, in turn this may 
increase patient safety [17]. There is also some evidence 
that decision support may improve the care process in 
chronic disease management and patient health [31]. 
Also, decision support can modify behavior of physi- 
cians in ordering tests [32]. These features add value to 
the current situation and might consequently lead to 
gains in job performance. 

Attributes on the organizational level 

On the level of the organization we also included three 
attributes in the DCE. They are related to practical sup- 
port, managerial support coherence and added value. 

The attribute technical support has the following 
levels: 

IT helpdesk: the regular hospital-wide helpdesk can be 
called for all kinds of IT problems, including issues in 
EMR. 

Training: The hospital ensures that a certified trainer 
shows you in a one day training what the system can do. 
When subsequently using the system, you can use the 
regular helpdesk. 

A number of studies found that organizational and 
technical factors influence the implementation of EMRs 
[3,4,16,18,27,33]. Technical support covers the belief of 
the existence of an organizational and technical infra- 
structure that support the use of the system [23]. In 
Rogers' theory of diffusion, it is linked to the triability 
and observability of an innovation [2]. 

Lack of training and technical support hinders the 
EMR implementation [18]. Training in using the EMR 
has shown to affect nurses' attitude towards the EMR 
positively [34] . We therefore expect that when implemen- 
tation is flanked by training, the users are more willing to 
use the innovation. 

Managerial support in the organization is also expected 
to affect the implementation and was included with the 
following levels were presented in the DCE: 

Biding: The head of department emphasizes that the use 
of the EMR should interfere the regular work as little as 
possible. 

Stimulating: The head of department emphasizes the 
importance of good use of EMR for the quality of the 
work of the department. 

Managerial attitudes have shown to affect the success 
of the implementation in case of EMRs [18], but also in 
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other innovations [35,36] in several ways, for instance 
in information diffusion and by developing strategies 
regarding to innovations [23,37]. Research of Grol 
et al. show that implementations improve if managers 
have a stimulating attitude towards an innovation 
[5,6]. In case of EMR, it may be expected that a stimu- 
lating attitude of the head of department leads to more 
acceptance. 

The final attribute on organizational level relates to or- 
ganizational use of the information that becomes avail- 
able after implementation of the EMR 

This attribute consisted of the following levels: 

No overview: You do not receive any information from 
the EMR concerning your department. 

Monthly overview: You receive a monthly e-mail with 
an overview of your department with number of patients, 
mean age of patients, number of diagnoses, number of 
complications, and satisfaction ratings of patients. 

When the organization uses the innovation to give the 
employees systematic feedback on performance, that 
they did not have before the implementation, this is 
showed to be valued by clinicians [38-40]. We therefore 
expect that when the implementation of the EMR is ac- 
companied with systematic feedback on performance of 
the department, this will positively affect the willingness 
to use the EMR. 



Participants 

The respondents consisted of physicians (medical spe- 
cialists and medical residents) and nurses (head nurses, 
specialist nurses, regular nurses and other) of different 
hospitals in The Netherlands. Participants were derived 
from internet survey panels via external research agencies. 



311 nurses and 550 physicians were contacted to partici- 
pate. The respondents could fill out the questionnaire via 
the internet. Respondents who completed the question- 
naire received a monetary reward. 

According to The Dutch National Ethics Board (Central 
Committee on Research involving Human Subjects) formal 
testing by a medical ethical committee was not necessary 
because the responding clinicians were asked to fill out a 
1) single, anonymous survey that 2) did not include radical, 
incriminating or intimate questions and 3) filling out the 
survey did not require much time. This is in accordance 
with the guidelines laid down in the Declaration of 
Helsinki. 

Statistical analyses 

In nLogit 4.0 (Econometric Software Inc. 2007), Panel 
Mixed Multinomial Logit (Panel-MIXL) analyses were 
performed for the data of physicians and nurses separ- 
ately. This statistical method enables to adjust for the 
dependency of observations among individuals (i.e., every 
respondent answered multiple choice sets). Levels of attri- 
butes were effects coded because of their non-linearity. In 
contrast to dummy-coding, with effects-coding it is made 
possible to compare the different attributes with each 
other with respect to their relative importance [41]. The 
attribute estimates for the reference category (coded as -1) 
can be calculated by multiplying -1 with the estimate of 
the attribute (see Table 1 for coding). 

Alternative specific constants (ASCs) were included 
for alternative A and B. Since these constants signifi- 
cantly differed from each other in the physicians' data, 
no generic constant could be included. To perform 
uniform analyses, in analysis of nurses ASCs were used 
as well. 



Table 1 Attributes, coding in analyses, and link with hypotheses 



Attribute 


Attribute level and coding 


Level of implementation and 
aspect 


Data entry hardware; hardware to access EMR 


-1 Computer/workstation 


Innovation level, ease of use 




1 Tablet 




Flexibility of interface; user can tailor to wishes, versus static 


-1 Static: need to scrol 


Innovation level, ease of use 




1 Flexible: user is able to tailor set desired 






information 




Decision support in EMR or not 


-1 No decision support present 


Innovation level, added value 




1 Decision support present 




Practical support, regular IT helpdesk or supported by training 


-1 Regular IT support 


Organization level, practical 




1 IT support, combined with training 


support 


Attitude of your manager; stimulating or biding 


-1 Biding 


Organization level, managerial 




1 Stimulating 


support 


Performance feedback; monthly overview of performance 


-1 No performance feedback 


Organization level, added value 


department 


1 monthy overview performance department 
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The utility regression equations were described as 
following: 

U = (ASCC) + fiV (data entry by tablet) 
+ ^2* (technical support incl training) 
+ /S3* (attitude head of department stimulating) 
+ /S4* (performance feedback in EMR) 
+ /35* (flexibility high) 
+ /S6* (decision support in EMR) 

U (opt-out) = 0 

The need for random parameters were tested in this 
model based on model fit. We tested whether including 
different attributes as random parameter (assuming nor- 
mal distribution) resulted in a significant better model 
fit based on AIC and log-likelihood. The final model 
contained no random parameters, except for the intercept. 
The answers were coded as opt-out when respondents 
had answered that that would prefer not to use the EMR 
in either scenario of the choice set. 

Outcomes of analyses were different attribute esti- 
mates (beta values), corresponding standard errors and 
p-values calculated with the use of Z-tests, used signifi- 
cance level was 0.05. Negative estimates imply that re- 
spondents prefer the reference level of the attribute. 
With these attribute estimates a ranking of attributes 
was made, from high (highest relative importance) to 
low (lowest relative importance). 



Results 

Sample 

In total, 148 of 311 nurses (47.6%) and 150 of 550 physi- 
cians (27.3%) completed the questionnaire, Table 2 presents 
the descriptive statistics of the sample. Gender distribution 
and mean ages of the sample were compared with popula- 
tion data obtained by Statistics Netherlands [41]. Mean 
age of male nurses (16% of all nurses in this study) was 

42.5 (s.d. = 11.6), female nurses (84%) had a mean age of 
40.1 (s.d. = 11.7). Gender distribution was comparable to 
nurses' population in the Netherlands (15% male versus 
85% female), only mean ages were somewhat lower in our 
study population. In Dutch population of nurses in 2010, 
mean ages were 47.7 for male nurses and 43.4 for female 
nurses [41,42]. 

For the 150 physicians our study, mean age for men 
was 48.3 (SD 9.6) in our study, which was corresponding 
with the average age of Dutch population of male physi- 
cians (48.4). Mean age for female physicians was a little 
higher in our study, namely 44.4 (SD 8.2) compared to 

40.6 in Dutch population of female physicians. Gender 
distribution of the physicians in the study differed from 
the national population. In this study there were rela- 
tively more male physicians that did respond to the sur- 
vey compared to the overall physicians' population of 
2010 (54% male versus 46% female physicians) [41]. 
When performing stratified analyses (not presented), 
the relative importance of attributes was not different 
between the genders. 



Table 2 Descriptive statistics of respondents 



Nurses (n = 148) 



Physicians (n = 150) 



Mean age, year (SD) 

Number of years working in current job 

Reported type of hospital 

University 

General hospital with teaching facilities 
General without teaching facilities 
Peripheral 

ndependent treatment center 

Other 

Profession 

Medical specialist 
Medical Resident 
Head Nurse 
Nurse 

Nurse specialist 
Other 



Male (n = 23) 
(16%) 

42.52 (11.58) 
9.71 (9.61) 

5 
5 
9 
0 
0 
4 



0 
15 
6 
2 



Female (n = 125) 
(84%) 

40.14 (11.71) 



26 
32 
55 
1 

2 
9 



10 
88 
24 
3 



Male (n = 109) 
(73%) 

48.26 (9.64) 

1273 (9.05) 

19 

40 

45 

0 

3 

2 

103 
6 



Female (n = 41) 

(27%) 

43.90 (7.72) 



1 
2 
0 

38 
3 



The category 'other' exists of a consulting assistant and employees of surgery department, anesthesia, and general support function. 
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Respondents were working in different types of hospi- 
tals, i.e., university hospitals, general hospitals, and inde- 
pendent treatment centers. Nurses that participated in this 
study were mainly regular nurses or nurse specialists. The 
group of physicians consisted of medical specialists and 
residents. 

Representativeness of study population was checked 
on gender distribution, age, profession and reported type 
of hospital. Based on the information presented above, 
we concluded that our population was a good reflection 
of overall population of nurses. We were able to include 
relatively fewer female physicians compared to the national 
physicians' population. However, tests reveal no systematic 
different answer patterns between the genders. 

Outcomes of Panel-MIXL analyses were estimates for 
the different attributes. With these attribute estimates a 
ranking of attributes was made, from high (highest relative 
importance) to low (lowest relative importance). 

Analyses 

Results of the Panel-mixed model are presented in Table 3 
(physicians) and Table 4 (nurses). In both models the 
intercepts were set random, controlling for the panel 
structure of the data and all predictor variables were 
fixed. For the statistically significant parameters, the 
size of the parameter indicates the relative importance 
in the preference of the respondents. The nurses chose 
opt-out in 23.2% of their answers, the physicians chose 
opt-out in 21.4% of their answers. The intercept in the 
model of the nurses is positive and significant (Table 4), 
and in the model of the physicians (Table 3) the intercept 
is positive but not significant. The sd's of the intercept are 
not significant, but we see that the size of the sd for the 
physicians (Table 3) is larger than in the model of the 
nurses. This means that the nurses had a smaller variance 
in their answers than did the physicians. 

A high flexibility of interface of the EMR turns out to 
have highest relative importance in both analyses. In sam- 
ple of physicians, decision support was the second largest 
beta, whereas for the nurses the stimulating attitude by 
the head of department had the second largest beta. 



Subsequently, performance feedback in the system was 
found to be important. Attribute estimates were negative 
for data entry hardware and technical support, which 
means that respondents preferred workstation above tab- 
let as data entry hardware and, respondents preferred just 
to get support of the IT helpdesk above an additional 
training. Technical support and data entry hardware were 
perceived by physicians and nurses as relatively least im- 
portant. In both analyses technical support did not yield 
significant results. 



Discussion 

This study investigated the preference structure towards 
the use of EMRs in the daily practice of work in a hospital 
of nurses and physicians by quantifying the relative im- 
portance of several aspects in the use of the EMR. Because 
physicians and nurses work in different situations in hos- 
pitals, we expected to find differences in their preference 
structures. In this study we both found similarities and dif- 
ferences in relative importance of attributes perceived by 
physicians and nurses. These differences can be under- 
stood from the different positions in the hospitals. The 
jobs of nurses are more integrated in the organization's 
bureaucratic structure, whereas physicians work more au- 
tonomously, possibly explaining the expected result that 
the attitude of the head of department was more import- 
ant to the nurses than to the physicians. 

The three most prominent factors that the physicians 
responded to were related to ease of use and added value 
of the innovation. The first two (flexible interface and deci- 
sion support) are innovation-level factors and need to be 
addressed by the producers of the EMR. The third, feed- 
back on performance, can be organized by the hospital. 

Even though the results of the nurses show many simi- 
larities with those of the physicians, it also shows that 
nurses attach more value to the organizational support 
in their preference to use an innovation. After consider- 
ing the ease of use, managerial support was considered 
in their preference to use the EMR. After this, the added 
value of the innovation was considered. 



Table 3 Results of panel-MIXL analyses, for the physicians; dependent variable is chosen scenario 


Attribute 


Levels (-1 value vs 1 value) 


Parameter 


s.e. 


Flexible interface 


Need to scroll vs Personal tailor option 


0.43 


0.04** 


Decision support present 


No vs Yes 


0.25 


0.04** 


Feedback performance 


No feed back versus monthly feed back 


0.20 


0.04** 


Attitude head of department 


Biding vs Stimulating 


0.18 


0.04** 


Practical support, including training 


Regular helpdesk vs Helpdesk and training 


-0.10 


0.01** 


Data entry hardware 


Workstation vs Tablet 


-0.10 


0.04* 


ntercept 




0.67 


0.39 


Sd intercept 




0.87 


0.93 



*p< 0.01; *p< 0.05. 
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Table 4 Results of panel-MIXL analyses, for the nurses; dependent variable is chosen scenario 


Attribute 


Levels (-1 value vs 1 value) 


Parameter 


s.e. 


Flexible interface 


Need to scroll vs Personal tailor option 


0.35 


0.03" 


Attitude head of department 


Biding vs Stimulating 


0.23 


0.04** 


Decision support present 


No vs Yes 


0.22 


0.04** 


Feedback performance 


No feed back versus monthly feed back 


0.21 


0.04** 


Practical support 


Regular helpdesk vs Helpdesk and training 


-0.05 


0.04 


Data entry hardware 


Workstation vs Tablet 


-0.17 


0.04** 


Intercept 




0.38 


0.10** 


Sd intercept 




0.13 


1.06 



p < 0.01; *p< 0.05. 



Implications for the implementation practices is that 
for physicians the innovation need to be fully developed 
and have immediate ease of use and added value, while 
for the nurses, the managerial support can positively 
affect the implementation process amongst the users. 

The shared preference of nurses and physicians of a 
flexible interface may be interpreted as a signal that user 
friendliness of the current systems may be improved. 
The finding in this study is in line with previous research 
[18,27], and with finding of the Health Care Inspectorate 
that reported that crucial information is not always imme- 
diately retrievable, possibly because of hard to read EMR. 
The inspectorate advises that the information need of 
users plays a central role in the development of EMRs 
[43] . That this advice is still applicable is also echoed in an 
essay question that included in the questionnaire that con- 
tained the current DCE. A majority of respondents that 
had used to opportunity to answer this question, referred 
to the cumbersome nature of their EMR that causes ineffi- 
ciency and inconvenience. 

Decision support was also found to be important by 
both groups. Decision support warns users of EMRs for 
risky situations. Currently, decision support is not often 
an integral part of EMR. If the expectations by Gartner 
on future EMRs become true, somewhere in the future, 
this feature will become standard. Based on the out- 
comes in this study, we expect that generation of EMRs 
to be embraced more heartily by the users than the 
current generation. 

Both physicians and nurses preferred monthly per- 
formance feedback above no feedback, which was ranked 
third in both groups. With this monthly overview of pa- 
tient characteristics, number of diagnoses, complications 
and patient satisfaction scores, it is possible to motivate 
physicians and nurses change their usual practice when 
they receive reports of inferior or inconsistent care [38]. 
Consequently, this will result in an improvement of quality 
of care, delivered by that physician or nurse, which also 
affects the overall quality of health care. Physicians do 
appreciate this attribute; this might imply that they are 



willing to improve the quality of delivered care. This is a 
potentially useful outcome for policy makers who are 
involved in improving health care, but also for imple- 
menters of the EMR. A performance feedback system 
can make clinicians aware of their delivered quality of 
care and thereby possibly contribute to the improve- 
ment of quality of care. 

Against expectations, nurses and physicians preferred 
the use of computers or workstations above tablets. This 
contrasts with evidence in literature suggesting that 
nurses and physicians prefer the use of tablets instead of 
workstations, because of various reasons. On the one 
hand, tablet use may enhance the patient-caregiver inter- 
action when doctors are not sitting behind a desk watch- 
ing the computer screen, but use a hand-held device 
when consulting patients [28]. But on the other hand, 
another study reported that smartphone and tablet use 
in hospital settings may have several negative effects, 
such as distraction by text messaging, unprofessional be- 
havior, infection or hygiene risks, and interference with 
other medical equipment [44]. It is unclear why respon- 
dents in our study did not prefer the use of tablets. A 
potential cause can be the fact that clinicians are happy 
in the current situation where they are working with 
computer or workstations. Many EMR contain multiple 
tabs, and this may be easier to operate from a desktop 
computer. Ease of use may also apply to writing add- 
itional information in text, because writing text with a 
tablet is less comfortable than typing on a keyboard. 
Choosing for the tablet for data entry means a new tech- 
nique and thus a change of behavior and higher costs of 
the innovation. When using the 'old-fashioned' worksta- 
tions for data entry in the EMR, users do not need to 
dramatically change their behavior, which leads to lower 
costs compared to when using Tablets. 

The second unexpected result is finding that the re- 
spondents do not prefer to get training for the EMR. 
Several studies have shown the importance of training 
and technical support [18,27,33,34]; training in using the 
system has shown to influence the attitude of nurses 
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positively [34]. Also, training in the system and adequate 
technical support might be necessary to counteract re- 
sistance of using EMRs [18,27,42]. In the scenario, we 
stated that the training would take a day. This may have 
generated a response we did not anticipate in advance. It 
implies that the added value of the training costs a day 
of valuable time, maybe even spare time, because this 
was not clearly explicated in the questionnaire. We as- 
sume that the wording in our design is responsible for 
the relatively low priority given to this factor. 

Concerning the use of DCE's in implementation stud- 
ies, a number of conclusions can be drawn after this 
study. Previous studies lead to contradictory conclusions 
on using DCEs in implementation research. The most 
critical of these studies stated that DCEs were less suited 
to elicit preferences from physicians based on a response 
rate of 10% [13]. The response rate in this study was 
27% for physicians and 48% for nurses. This means that 
we also found that physicians are harder to recruit, how- 
ever, in the description of our sample we presented 
mean age and percentages of genders. In the nurses 
sample, we found no substantial deviations between the 
national statistics and our sample. The sample contained 
fewer female physicians than in the national population, 
however we found no evidence that female respondent 
physicians had structurally different answer patterns. 
This under representation of female physicians could be 
because female physicians are less interested in the topic 
of research. We cannot exclude that on average the female 
physicians had answered differendy if they were would 
have been included. A second indication of less added 
value of the used methods would be that respondents 
stopped seriously answering the questions and answered 
the questions without reading them. This may have shown 
up in respondents answering the scenario on the right side 
(or left side) of the screen. However, we tested for this left 
to right bias and found no evidence of this. Third we 
studied the answers of the respondents to the final 
question in the questionnaire whether they had com- 
ments on the questionnaire. We found there that two 
physicians and five nurses objected to this way of ques- 
tioning. On the other hand, two nurses praised the 
questionnaire. Finally, the plausibility and understand- 
ability of the majority of the answer patterns lead us to 
conclude that this DCE has generated valuable insights 
in the preferences of physicians and nurses and that this 
method can also be considered as a tool to study the 
preferences of users for other innovations. 

The strong preference for the flexible interface for 
both user groups may be a signal that the reported pref- 
erences are influenced to the moment of answering the 
DCE. Given the findings of earlier research and of the 
inspectorate, many of the EMRs currently used in Dutch 
hospitals are not optimally suited to give health care 



professionals a quick and easy access to the information 
that they need. The current study asked the respondents 
to refer to their own situation. The importance that they 
attach to the interface may therefore be caused by 
current frustrations of their own EMR. If this study will 
be replicated later on, when a new generation of EMRs 
is implemented, we expect the order of relevance to be 
different. The fact that we could not correct for the fact 
that implementation stage is likely to affect the prefer- 
ences of the users [45], is a limitation of this study. 

In this study, internet panels were used to approach 
respondents. Quality of responses of internet panels were 
already described by Chang and Krosnick, internet panel 
members will give more reliable answers on attitude ques- 
tions when they become more experienced in responding 
to them [46]. Nonetheless, experienced completing of 
questionnaires may also lead to 'panel conditioning.' This 
implies that an increasing experience in doing surveys 
makes members less representative of the population they 
were derived from. However, no evidence or only very 
small effects were found when studying this subject. Also, 
sample composition bias caused by interest in the topic 
may be present, because those who are interested in the 
topic might be more likely to participate in the survey 
[47,49]. However, this problem may be unavoidable in 
every form of research that depends on respondent 
participation. 

Conclusions 

To our knowledge, this is the first study where relative 
importance of barriers and facilitators of EMRs in hospi- 
tals are quantified among clinicians. DCEs are used in 
other fields of research, but it is relatively unknown in 
health care based implementation literature. However, 
prioritizing the barriers and facilitators for other types of 
interventions may give valuable information for people 
designing implementation strategies. Although it is a snap- 
shot in time, policymakers, implementers and developers 
of the EMR may use the information to fine tune imple- 
mentation processes and anticipate the preferences of the 
users and future implementation processes. This study 
shows that for the nurses, managerial support will be more 
effective in promoting acceptance of the EMR than for 
physicians. It also shows that the physicians are mainly 
concerned with the ease of use and added value of the 
innovation. This means that during the implementation 
the EMR has to function flawlessly directly and should 
be easy to work with, in order to gain support from the 
physicians. The hospital will positively affect the sup- 
port of the physicians if they use the information of the 
EMR for performance feedback on departmental level. 
For the physicians, the key to innovation acceptance lies 
within added value and ease of use, while for the nurses 
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the organizational context can help to affect their ac- 
ceptance for the innovation. 

Endnotes 

a Al though a DCE and conjoint approach differ meth- 
odologically [14,47], they both address the same type of 
questions and they both enable eliciting the relative 
importance of included attributes. 
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Additional file 2: Instruction sheet for respondents. 



Abbreviations 

DCE: Discrete choice experiment; EMR: Electronic medical record; 
IT: Information technology. 

Competing interests 

All authors declare to have to competing interests 
Authors' contributions 

MHLS wrote the concept of the main body of the text and contributed to 
the analyses. FK contributed in the conceptual work of the text and critically 
reviewed text and analyses. AJS contributed in the conceptual work of the 
text and critically reviewed text and analyses. RN contributed to the 
acquisition of the data and reviewed the text. JV contributed to the design 
and interpretation of the analyses. MSL developed the research framework 
and data collection, supervised data acquisition, co-wrote the main body of 
the text and conducted analyses. All authors read and approved the final 
manuscript. 

Author details 

'Health Sciences, specialization Policy and Organization of Health Care, VU 
University Amsterdam, Amsterdam, The Netherlands, department of 
Sociology, Erasmus University Rotterdam, Rotterdam, The Netherlands, 
department Quality of Care and Health Economics, National Institute for 
Public Health and the Environment (RIVM), Center for Nutrition, Prevention 
and Health Services, Bilthoven, The Netherlands. 4 University Medical Center 
Utrecht, Julius Center for Health Sciences and Primary Care, Utrecht, The 
Netherlands. 5 National Institute for Public Health and the Environment 
(RIVM), Bilthoven, The Netherlands. 

Received: 1 1 February 2014 Accepted: 27 May 2014 
Published: 5 June 2014 

References 

1 . Bramble JD, Gait KA, Siracuse MV, Abbott AA, Drincic A, Paschal KA, Fuji KT: 
The relationship between physician practice characteristics and 
physician adoption of electronic health records. Health Care Manage Rev 

2010, 35:55-64. 

2. Rogers EM: Diffusion of Innovations. 4th edition. New York: Simon and 
Schuster; 1 995. 

3. Chaudoir SR, Dugan AG, Barr CHI: Measuring factors affecting 
implementation of health innovations: a systematic review of structural, 
organizational, provider, patient, and innovation level measures. 

Implement Sci 2013, 8:22. 

4. Fleuren M, Wiefferink K, Paulussen T: Determinants of innovation within 
health care organizations: literature review and Delphi study. Int J Qual 
Health Care 2004, 16:107-123. 

5. Grol R, Grimshaw J: Research into practice I From best evidence to best 
practice : effective implementation of change in patients care. Lancet 
2003, 362:1225-1230. 

6. Grol R, Wensing M: What drives change? Barriers to and incentives for 
achieving evidence-based practice. Med J Aust 2004, 180:S57-S60. 

7. Grol R, Wensing M: Implementatie, effectieve verbetering van de patientenzorg. 
Amsterdam: Reed Business; 201 1 . 



8. Davis FD, Bagozzi RP, Warshaw PR: User acceptance of computer 
technology: a comparison of two theoretical models. Manage Sci 1989, 
35:982-1003. 

9. de Bekker-Grob EW, Ryan M, Gerard K: Discrete choice experiments in 
health economics: a review of the literature. Health Econ 2010, 
21:145-172. 

10. Lancsar E, Louviere J: Conducting discrete choice experiments to inform 
healthcare decision making: a user's guide. Pharmacoeconomics 2008, 
26:661-677. 

11. Ryan M, Farrar S: Using conjoint analysis to elicit preferences for health 
care. BMJ 2000, 320:1530-1533. 

12. Farley K, Thompson C, Hanbury A, Chambers D: Exploring the feasibility of 
Conjoint Analysis as a tool for prioritizing innovations for 
implementation. Implementation Science 2013, 8:56. 

13. van Helvoort-Postulart D, van Der Weijden T, Dellaert BGC, de Kok M, vM 
MF, Dirksen CD: Investigating the complementary value of discrete 
choice experiments for the evaluation of barriers and facilitators in 
implementation research: a questionnaire survey. Implementation Science 
2009, 4:10. 

14. Goldstein DH, Phelan R, Wilson R, Ross-White A, Vandenkerkhof EG, Penning 
JP, Jaeger M: Brief review: Adoption of electronic medical records to 
enhance acute pain management. Can J Anaesth 2013, 61 (2):1 64—1 79. 

15. Chaudhry B, Wang J, Wu S, Maglione M, Mojica W, Roth E, Morton SC, 
Shekelle PG: Systematic review: impact of health information technology 
on quality, efficiency, and costs of medical care. Ann Intern Med 2006, 
144:E-12-E-22. 

16. Nambisan P, Kreps Gt, Polit S: Understanding electronic medical record 
adoption in the United States: communication and sociocultural 
perspectives. Interact J Med Res 2013, 2:e5. 

17. Hillestad R, Bigelow J, Bower A, Girosi F, Meili R, Scoville R, Taylor R: Can 
electronic medical record systems transform health care? Potential 
health benefits, savings, and costs. Health Aff (Millwood) 2005, 
24:1103-1117. 

18. Boonstra A, Broekhuis (vl: Barriers to the acceptance of electronic medical 
records by physicians from systematic review to taxonomy and 
interventions. BMC Health Serv Res 2010, 10:231. 

19. Bridges JFP, Hauber aB, Marshall D, Lloyd A, Prosser L, Regier D, Johnson FR, 
Mauskopf J: Conjoint analysis applications in health-a checklist: a report 
of the ISPOR good research practices for conjoint analysis task force. 
Value Health 201 1, 14:403-413. 

20. Ryan M, Gerard K, Amaya-Amaya M: Using Discrete Choice Experiments to 
Value Health and Health Care. In e Economics of Non-Market Goods and 
Resources, Dordrecht: Springer. Edited by Bateman IJ. Dordrecht, 
Netherlands: Springer; 2008. 

21 . Manski CF: The structure of random utility models. Theory Decis 1977, 
8:229-254. 

22. Bliemer MCJ, Rose JM: Efficiency and Sample Size Requirements for 
Stated Choice Experiments. In Transportation Research Board 88th Annual 
Meeting; 2009. 2009. 

23. Venkatesh V, Morris MG, Davis GB, Davis FD: User acceptance of 
information technology: toward a unified view. MIS Q 2003, 27:425-478. 

24. Davis FD: A technology acceptance model for empirically testing new 
end-user information systems: Theory and results. Cambridge: 
Massachusetts Institute of Technology, Thesis; 1986. 

25. Davis FD: Perceived usefulness, perceived ease of use, and user 
acceptance of information technology. MIS Q 1989, 13:319-340. 

26. Venkatesh V, Davis FD: A theoretical extension of the technology 
acceptance model: four longitudinal field studies. Manage Sci 2000, 
46:186-204. 

27. McGinn CA, Grenier S, Duplantie J, Shaw N, Sicotte C, Mathieu L, Leduc Y, 
Legare F, Gagnon MP: Comparison of user groups' perspectives of 
barriers and facilitators to implementing electronic health records: a 
systematic review. BMC Med 201 1, 9:46. 

28. Rao AS, Adam TJ, Gensinger R, Westra BL: Study of the factors that 
promoted the implementation of electronic medical record on iPads at 
two emergency departments. AMIA Annu Symp Proc 2012, 2012:744-752, 

29. Sclafani J, Tirrell TF, Franko 01: Mobile tablet use among academic 
physicians and trainees. J Med Syst 201 3, 37:9903. 

30. Darbyshire P: 'Rage against the machine?': nurses' and midwives' 
experiences of using computerized patient information systems for 
clinical information. J Clin Nurs 2004, 13:1 7-25. 



Struik et al. Implementation Science 2014, 9:69 
http://www.implementationscience.eom/content/9/1/69 



Page 11 of 1 1 



31. Roshanov PS, Misra S, Gerstein HC, Garg AX, Sebaldt RJ, Mackay J, Weise- 
Kelly L, Navarro T, Wilczynski NL, Haynes RB: Computerized clinical decision 
support systems for chronic disease management: a decision-maker- 
researcher partnership systematic review. Implement Sciences 201 1, 6:92. 

32. Roshanov PS, You JJ, Dhaliwal J, Koff D, Mackay J, Weise-Kelly L, Navarro T, 
Wilczynski NL, Haynes RB: Can computerized clinical decision support 
systems improve practitioners' diagnostic test ordering behavior? A 
decision-maker-researcher partnership systematic review. Implementation 
Sci 201 1,6:88. 

33. Ludwick D, Manca D, Doucette J: Primary care physicians' experiences 
with electronic medical records Implementation experience in 
community, urban, hospital, and academic family medicine. Can Fam 
Physician 2010, 56:40-47. 

34. Sassen EJ: Love, hate, or indifference: how nurses really feel about the 
electronic health record system. Comput Inform Nurs 2009, 27:281-287. 

35. Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O: Diffusion of 
innovations in service organizations: systematic review and 
recommendations. Milbank 0 2004, 82:581-629. 

36. Jones RA, Jimmieson NL, Griffiths A: The impact of organizational culture 
and reshaping capabilities on change implementation success: the 
mediating role of readiness for change. J Manage Stud 2005, 
42(2):361-386. 

37. Birken SA, Lee S-YD, Weiner BJ: Uncovering middle managers' role in 
healthcare innovation implementation. Implement Science 2012, 7:28. 

38. van der Veer S: Systematic quality improvement in healthcare: clinical 
performance measurement and registry-based feedback Amsterdam: 
University of Amsterdam, Faculty of Medicine; 201 2. 

39. Venkatesh V: Determinants of perceived ease of use: integrating control, 
intrinsic motivation, and emotion into the technology acceptance 
model. Info Syst Res 2000, 1 1:342-365. 

40. Sapyta J, Riemer M, Bickman L: Feedback to clinicians: theory, research, 
and practice. J Clin Psychol 2005, 61:145-153. 

41 . CBS: StatLine- Medically qualified, employment status, position in employment, 
by occupation. In Book StatLine- Medically qualified, employment status, position in 
employment, by occupation Den Haag: Statistics Netherlands (CBS); 201 3. [http// 
statline.cbs.nl/StatWeb/publication/?DM=SLNL&PA=81551NED&Dl=0,2,9,14,19- 
20&D2=1-2&D3=M0&D4=4,65&D5=I&HDR=T&STB=G3,G1,G4,G2&VW=T]. 

42. Ludwick D, Doucette J: Adopting electronic medical records in primary 
care: lessons learned from health information systems implementation 
experience in seven countries. Int J Med Info 2009, 78:22-31. 

43. IGZ (Health Care Inspectorate): State of Health care 201 1, Information 
exchange in health care: IT does not solve problems without 
standardization of information exchange. In Book State of Health care 

201 1, Information exchange in health care: IT does not solve problems without 
standardization of information exchange. Utrecht: IGZ, Health Care 
Inspectorate; 201 1 . 

44. Gill PS, Kamath A, Gill TS: Distraction: an assessment of smartphone usage 
in health care work settings. Risk Manage Healthcare Policy 2012, 
5:105-114. 

45. Plmejad H, Niazkhani Z, van der Sijs H, Berg M, Bal R: Evaluation of the 
impact of a CPOE System on Nurse-physician communication. Meth Info 
Med 2009, 48:350-360. 

46. Chang L, Krosnick J: National Surveys Via Rdd Telephone Interviewing 
Versus the Internet: Comparing Sample Representativeness and 
Response Quality. Public Opin 0 2009, 73:641-678. 

47. Louviere JJ, Flynn TN, Carson RT: Discrete choice experiments are not 
conjoint analysis. J Choice Model 2010, 3:57-72. 

■ > 

doi:1 0.1 1 86/1 748-5908-9-69 

Cite this article as: Struik et at: The preferences of users of electronic 
medical records in hospitals: quantifying the relative importance of 
barriers and facilitators of an innovation. Implementation Science 
2014 9:69. 



f \ 

Submit your next manuscript to BioMed Central 
and take full advantage of: 

• Convenient online submission 

• Thorough peer review 

• No space constraints or color figure charges 

• Immediate publication on acceptance 

• Inclusion in PubMed, CAS, Scopus and Google Scholar 

• Research which is freely available for redistribution 



Submit your manuscript at /^'\ n! _, ul _-| r Q r,tr=l 

www.biomedcentral.com/submit ammBa central 



